A Study on Trust Region Update Rules in Newton Methods for Large-scale Linear Classification

Œe main task in training a linear classi€er is to solve an unconstrained minimization problem. In applying an optimization method to obtain a model, typically we iteratively €nd a good direction and then decide a suitable step size. Past developments of extending optimization methods for large-scale linear classi€cation focus on €nding the direction, but liŠle aŠention has been paid on adjusting the step size. In this work, we explain that inappropriate step-size adjustment may lead to serious slow convergence. Among the two major methods for step-size selection, line search and trust region, we focus on investigating the trust region methods. A‰er presenting some detailed analysis, we develop novel and e‚ective techniques to adjust the trust-region size. Experiments indicate that our new seŠings signi€cantly outperform existing implementations for large-scale linear classi€cation.